Dialog systems are continually evolving to handle less constrained spoken input, interpret user intent, and engage in natural dialog to accomplish complex tasks. Wizard systems using one or more human “wizards” may be used to simulate a fully automated experience in a dialog system to assist in early user studies and data collection before the dialog system is completed. For example, when simulating a speech system, a human wizard may be used behind the scenes to transcribe the user's speech rather than using an actual speech recognition system.